Image processing apparatus and image processing method

ABSTRACT

An image processing apparatus that an image input unit; a parallax acquisition unit configured to acquire a per-pixel or per-region parallax between two-viewpoint images; a main subject detection unit configured to detect a main subject on the two-viewpoint images; a parallax acquisition unit configured to acquire a parallax of the main subject; a setting unit configured to set a conversion factor of the parallax; a correction unit configured to correct the conversion factor of the parallax per pixel, per region, or per image; a multi-viewpoint image generation unit configured to convert at least one image of the two-viewpoint images in accordance with the corrected conversion factor of the parallax; an image adjustment unit configured to shift the two-viewpoint images or multi-viewpoint images to obtain a parallax appropriate for stereoscopic view; and a stereoscopically-displayed image generation unit configured to generate a stereoscopically-displayed image.

TECHNICAL FIELD

The present invention relates to an image processing apparatus and animage processing method, and more particularly, to an image processingapparatus and an image processing method which generate multi-viewpointimages from inputted two-viewpoint images.

BACKGROUND ART

A technology of generating an image corresponding to an arbitraryintermediate viewpoint from two images which have viewpoints differentfrom each other and are photographed as a stereo image is important todisplay an appropriate stereoscopic image on a stereoscopic photographicprint having a surface on which a lenticular lens sheet is attached orvarious other stereoscopic image display devices.

PTL 1 describes a technology in which a depth or an increase in parallaxbetween both eyes is linearly compressed with respect to an arbitrarypoint as the center, and a desired depth effect can be accordinglyobtained through simple conversion. According to this technology, adepth effect in a predetermined region of a three-dimensionallydisplayed image to be generated can be changed, which thus enables aflexible response to a preference of a viewer and the like.

CITATION LIST Patent Literature

-   {PTL 1} Japanese Patent Application Laid-Open No. 08-331607

SUMMARY OF INVENTION Technical Problem

However, according to the invention described in PTL 1, since control ona target value of a stereoscopic effect of a main subject is notperformed, in the case where a process of suppressing a parallax isperformed on an image which cannot be stereoscopically viewed due to anexcessively large parallax, there arises a disadvantage as an adverseeffect that parallax distribution of the main subject becomesexcessively small, so that the stereoscopic effect may be lost.

The present invention has been made in view of the above-mentionedcircumstances, and therefore has an object to provide an imageprocessing apparatus and an image processing method which generatemulti-viewpoint images in which a parallax of an entire image issuppressed to be small enough to enable stereoscopic view while astereoscopic effect of a main subject is maintained.

Solution to Problem

In order to achieve the above-mentioned object, an image processingapparatus according to a first aspect of the present invention includes:an image input unit configured to receive two-viewpoint images having aparallax therebetween; a parallax acquisition unit configured to acquirea per-pixel or per-region parallax between the two-viewpoint images; amain subject detection unit configured to detect a main subject on thetwo-viewpoint images; a parallax acquisition unit configured to acquirea parallax of the main subject; a setting unit configured to set aconversion factor of the parallax in accordance with respectiveviewpoint positions of multi-viewpoint images to be generated; acorrection unit configured to correct the conversion factor of theparallax per pixel, per region, or per image on a basis of the parallaxof the main subject; a multi-viewpoint image generation unit configuredto convert at least one image of the two-viewpoint images in accordancewith the corrected conversion factor of the parallax, to therebygenerate the multi-viewpoint images; an image adjustment unit configuredto shift the two-viewpoint images or the multi-viewpoint images in ahorizontal direction so that the parallax of the main subject on themulti-viewpoint images becomes a parallax appropriate for stereoscopicview; and a stereoscopically-displayed image generation unit configuredto generate a stereoscopically-displayed image on a basis of themulti-viewpoint images.

According to the first aspect, the conversion factor of the parallaxaccording to the respective viewpoint positions of the multi-viewpointimages to be generated is corrected per pixel on the basis of theparallax of the main subject, and hence it is possible to generate themulti-viewpoint images in which the parallax of the entire image issuppressed to be small enough to enable stereoscopic view while thestereoscopic effect of the main subject is maintained.

The image processing apparatus according to a second aspect of thepresent invention further includes, in the first aspect, a holding unitconfigured to hold target parallax distribution of an entire image andtarget parallax distribution of the main subject, wherein: the settingunit sets the conversion factor of the parallax so that parallaxdistribution of the entire image of the multi-viewpoint images to begenerated satisfies the target parallax distribution of the entireimage; and the correction unit corrects the conversion factor of theparallax per pixel, per region, or per image so that parallaxdistribution of the main subject on the multi-viewpoint images to begenerated satisfies the target parallax distribution of the mainsubject.

This makes it possible to generate the multi-viewpoint images having theparallax distribution of the entire image and the parallax distributionof the main subject which are appropriate.

The image processing apparatus according to a third aspect of thepresent invention further includes, in the first or second aspect, anoutput unit configured to output the generatedstereoscopically-displayed image at a predetermined size; and amodification unit configured to modify the target parallax distributionof the entire image and the target parallax distribution of the mainsubject in accordance with the predetermined size.

This makes it possible to generate the multi-viewpoint images having theparallax according to the output size.

In the image processing apparatus according to a fourth aspect of thepresent invention, the correction unit corrects the conversion factor ofthe parallax per pixel, per region, or per image on a basis of adifference between the per-pixel or per-region parallax and the parallaxof the main subject, in the first to third aspects.

This makes it possible to generate the multi-viewpoint images having theparallax of the entire image and the parallax of the main subject whichare appropriate.

In the image processing apparatus according to a fifth aspect of thepresent invention, the correction unit corrects the conversion factor ofthe parallax so that the parallax is more suppressed in a pixel orregion having the larger difference between the parallaxes, in thefourth aspect.

This makes it possible to generate the multi-viewpoint images having theparallax of the entire image and the parallax of the main subject whichare appropriate.

In the image processing apparatus according to a sixth aspect of thepresent invention, the correction unit corrects the conversion factor ofthe parallax so that the parallax is more increased in a pixel or regionhaving the smaller difference between the parallaxes, in the fourthaspect.

This makes it possible to generate the multi-viewpoint images having theparallax of the entire image and the parallax of the main subject whichare appropriate.

The image processing apparatus according to a seventh aspect of thepresent invention further includes, in the fourth aspect, aphotographing mode detection unit configured to detect a photographingmode set at a time of photographing the two-viewpoint images, fromcollateral information of the two-viewpoint images; and a selection unitconfigured to select, in accordance with the detected photographingmode, whether the correction unit corrects the conversion factor of theparallax so that the parallax is more suppressed in a pixel or regionhaving the larger difference between the parallaxes or corrects theconversion factor of the parallax so that the parallax is more increasedin a pixel or region having the smaller difference between theparallaxes.

This makes it possible to generate the multi-viewpoint images having theparallax of the entire image and the parallax of the main subject whichare appropriate.

The image processing apparatus according to an eighth aspect of thepresent invention further includes, according to the first aspect, aparallax histogram acquisition unit configured to acquire a histogram ofthe per-pixel or per-region parallax, wherein the correction unitcorrects the conversion factor of the parallax per pixel or per regionso as to make parallax gradations constant in accordance withfrequencies in the histogram.

This makes it possible to generate the multi-viewpoint images having theparallax gradations which are constant.

The image processing apparatus according to a ninth aspect of thepresent invention further includes, in the first aspect, a photographedscene recognition unit configured to recognize a photographed scene onthe two-viewpoint images; and a main subject setting unit configured toset a main subject region in accordance with the photographed scene,wherein the correction unit corrects the conversion factor of theparallax per region or per pixel in accordance with a distance on theimage from the main subject region.

This makes it possible to set the main subject through simpleprocessing, and to generate the multi-viewpoint images having theparallax of the entire image and the parallax of the main subject whichare appropriate.

In the image processing apparatus according to a tenth aspect of thepresent invention, the photographed scene recognition unit includes aphotographing mode detection unit configured to detect a photographingmode set at a time of photographing the two-viewpoint images, fromcollateral information of the two-viewpoint images; the main subjectregion setting unit sets, in a case where the detected photographingmode is a person mode, a central region of a plurality of regionsobtained by vertically dividing the image, as the main subject region;and the correction unit corrects the conversion factor of the parallaxper region or per pixel in accordance with a distance in the horizontaldirection from the main subject region, in the ninth aspect.

This makes it possible to set the main subject through simpleprocessing, and to generate the multi-viewpoint images having theparallax of the entire image and the parallax of the main subject whichare appropriate.

In the image processing apparatus according to an eleventh aspect of thepresent invention, the photographed scene recognition unit includes aphotographing mode detection unit which detects a photographing mode setat a time of photographing the two-viewpoint images, from collateralinformation of the two-viewpoint images; the main subject region settingunit sets, in a case where the detected photographing mode is a scenerymode, a central region of a plurality of regions obtained byhorizontally dividing the image, as the main subject region; and thecorrection unit corrects the conversion factor of the parallax perregion or per pixel in accordance with a distance in the verticaldirection from the main subject region, according to ninth aspect.

This makes it possible to set the main subject through simpleprocessing, and to generate the multi-viewpoint images having theparallax of the entire image and the parallax of the main subject whichare appropriate.

In the image processing apparatus according to a twelfth aspect of thepresent invention, the photographed scene recognition unit includes anextraction unit configured to extract a short-distance view region, amiddle-distance view region, and a long-distance view region from thetwo-viewpoint images; and the correction unit corrects the conversionfactor of the parallax per region or per pixel in accordance with therespective extracted regions, in the ninth aspect.

This makes it possible to set the main subject through simpleprocessing, and to generate the multi-viewpoint images having theparallax of the entire image and the parallax of the main subject whichare appropriate.

In the image processing apparatus according to a thirteenth aspect ofthe present invention, the setting unit sets, in a case where themulti-viewpoint images to be generated are four or more, the conversionfactor of the parallax so that a difference between viewpoint positionsin a central portion is larger than a difference between viewpointpositions at both end portions, in the first to twelfth aspects.

This makes it possible to generate the multi-viewpoint images which canbe viewed in various patterns.

In the image processing apparatus according to a fourteenth aspect ofthe present invention, the correction unit corrects the conversionfactor of the parallax per pixel, per region, or per image so that theconversion factor of the parallax of one image of the two-viewpointimages becomes 0; and the multi-viewpoint image generation unit convertsthe image whose conversion factor of the parallax is 0, to therebygenerate the multi-viewpoint images, in the first to thirteenth aspects.

This makes it possible to use an actually photographed image with regardto an image at at least one end, of the multi-viewpoint images.

In the image processing apparatus according to a fifteenth aspect of thepresent invention, the correction unit corrects the conversion factor ofthe parallax per pixel, per region, or per image so that the conversionfactor of the parallax of an image at a central viewpoint position, ofthe multi-viewpoint images to be generated, becomes smallest, in thefirst to thirteenth aspects.

This makes it possible to generate the multi-viewpoint images which arenatural.

In order to achieve the above-mentioned object, an image processingmethod according to a sixteenth aspect of the present inventionincludes: an image input step of receiving two-viewpoint images having aparallax therebetween; a parallax acquisition step of acquiring aper-pixel or per-region parallax between the two-viewpoint images; amain subject detection step of detecting a main subject on thetwo-viewpoint images; a parallax acquisition step of acquiring aparallax of the main subject; a setting step of setting a conversionfactor of the parallax in accordance with respective viewpoint positionsof multi-viewpoint images to be generated; a correction step ofcorrecting the conversion factor of the parallax per pixel, per region,or per image on a basis of the parallax of the main subject; amulti-viewpoint image generation step of converting at least one imageof the two-viewpoint images in accordance with the corrected conversionfactor of the parallax, to thereby generate the multi-viewpoint images;an image adjustment step of shifting the two-viewpoint images or themulti-viewpoint images in a horizontal direction so that the parallax ofthe main subject on the multi-viewpoint images becomes a parallaxappropriate for stereoscopic view; and a stereoscopic image generationstep of generating a stereoscopically-displayed image on a basis of themulti-viewpoint images.

Advantageous Effects of Invention

According to the present invention, a parallax of an entire image can besuppressed to be small enough to enable stereoscopic view while astereoscopic effect of a main subject is maintained.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a flow chart illustrating a generation process formulti-viewpoint images according to a first embodiment.

FIG. 2A is a view illustrating an example of inputted images.

FIG. 2B is a view illustrating a main subject region.

FIG. 2C is a view illustrating a correspondence relation between pointson right and left viewpoint images.

FIG. 2D is a view illustrating a left viewpoint image L and a rightviewpoint image R after the right viewpoint image R is shifted.

FIG. 2E is a view illustrating multi-viewpoint images C0 to C5.

FIG. 2F is a view illustrating final multi-viewpoint images S0 to S5.

FIG. 3A is a graph illustrating a function Table (x) for setting acorrection amount of a per-pixel parallax.

FIG. 3B is a graph illustrating the function Table (x) for setting thecorrection amount of the per-pixel parallax.

FIG. 3C is a graph illustrating the function Table (x) for setting thecorrection amount of the per-pixel parallax.

FIG. 3D is a graph illustrating the function Table (x) for setting thecorrection amount of the per-pixel parallax.

FIG. 4A is a graph illustrating the function Table (x) for setting thecorrection amount of the per-pixel parallax.

FIG. 4B is a graph illustrating the function Table (x) for setting thecorrection amount of the per-pixel parallax.

FIG. 4C is a graph illustrating the function Table (x) for setting thecorrection amount of the per-pixel parallax.

FIG. 4D is a graph illustrating the function Table (x) for setting thecorrection amount of the per-pixel parallax.

FIG. 5 is a flow chart illustrating a process for setting Factor2according to a main subject.

FIG. 6 is a flow chart illustrating a generation process formulti-viewpoint images according to a third embodiment.

FIG. 7 is a flow chart illustrating a process for setting a conversionfactor Factor1 of the per-image parallax.

FIG. 8A is a graph illustrating parallax distribution.

FIG. 8B is a graph illustrating the parallax distribution.

FIG. 9 is a flow chart illustrating the process for setting theconversion factor Factor1 of the per-image parallax.

FIG. 10 is a flow chart illustrating a generation process formulti-viewpoint images according to a fourth embodiment.

FIG. 11A is a graph illustrating the parallax distribution.

FIG. 11B is a graph illustrating the correction amount Factor2.

FIG. 12 is a flow chart illustrating a generation process formulti-viewpoint images according to a fifth embodiment.

FIG. 13 is a flow chart illustrating the generation process for themulti-viewpoint images according to the fifth embodiment.

FIG. 14A is a view for describing equal division of the left viewpointimage.

FIG. 14B is a view for describing the equal division of the leftviewpoint image.

FIG. 15 is a flow chart illustrating a generation process formulti-viewpoint images according to a sixth embodiment.

FIG. 16 is a block diagram illustrating an image processing apparatus10.

DESCRIPTION OF EMBODIMENTS

Hereinafter, preferred embodiments of an image processing apparatus andan image processing method according to the present invention aredescribed with reference to the attached drawings.

First Embodiment

FIG. 1 is a flow chart illustrating a generation process formulti-viewpoint images according to a first embodiment, and FIG. 2 areviews each illustrating an image example for describing each process. Inthe present embodiment, a parallax between the multi-viewpoint imageswhich are generated so that a main subject within the image has asuitable stereoscopic effect is controlled.

First, a parallel two-eye stereo image is inputted (Step S1). Theparallel two-eye stereo image refers to two images (two-viewpointimages), that is, a left viewpoint image and a right viewpoint imagehaving viewpoints different from each other. Here, description is givenby taking as an example the case where a left viewpoint image L and aright viewpoint image R illustrated in FIG. 2A are inputted.

Next, a conversion factor Factor1 of a parallax per image to begenerated is set (Step S2). Here, the case where six-viewpoint images(six multi-viewpoint images) are generated from the left viewpoint imageand the right viewpoint image is discussed. Conversion factors Factor1[0] to Factor1 [5] of parallaxes of these six images are set to, forexample, 0, 0.2, 0.4, 0.6, 0.8, and 1.0, respectively.

It should be noted that the number of viewpoints of multi-viewpointimages to be generated is not limited to 6, and may be decided asappropriate in accordance with an output apparatus.

Next, a main subject is detected from the left viewpoint image L, and aregion of the detected main subject within the image is acquired (StepS3). Hereinafter, with the use of top left coordinates (x_(topleft),y_(topleft)) and bottom right coordinates (x_(bottomright),y_(bottomright)) of a rectangular region including the main subject, theregion of the detected main subject within the image is described as aregion (x_(topleft), y_(topleft))−(x_(bottomright), y_(bottomright)). Itshould be noted that a shape of the main subject region is not limitedto a rectangle.

With regard to the main subject, for example, in the case where a personor a face of a person is detected within the image, the person or theface of the person is defined as the main subject. Alternatively, afocus position at the time of photographing is acquired from collateralinformation of inputted image data, and a region having a parallaxaccording to a distance of the focus position may be defined as the mainsubject. Still alternatively, it is conceivable that a parallaxhistogram is created on the basis of parallaxes acquired per pixel, anda region having a parallax of an average value, a mode value, a medianvalue, or the like in frequency distribution is defined as the mainsubject. In this case, it is preferable to create the parallax histogramwith regard to, for example, a region excluding 10% of the periphery ofthe image. This is because the main subject can be considered not toexist in the periphery of the image.

In the example illustrated in FIG. 2B, a face of a person is detected asthe main subject. In addition, a frame within the image illustrated inFIG. 2B indicates the region (x_(topleft),y_(topleft))−(x_(bottomright), y_(bottomright)) of the detected mainsubject.

Next, with reference to the left viewpoint image L, a per-pixel parallaxD (x, y) to the right viewpoint image R is acquired (Step S4). Theper-pixel parallax D (x, y) is acquired by calculating a correlationbetween the left viewpoint image L and the right viewpoint image R, tothereby detect a per-pixel corresponding point therebetween.

FIG. 2C illustrates a point L (x_(L), y_(L)) within the main subject onthe left viewpoint image L and its corresponding point R (x_(R), y_(R))on the right viewpoint image R. Here, y_(L)=y_(R), and the parallax atthis pixel is expressed by the parallax D (x, y)=x_(R)−x_(L).

Further, a parallax D [main] of the main subject is acquired on thebasis of the per-pixel parallax D (x, y) (Step S5). The parallax D[main] of the main subject is expressed as an average value of theper-pixel parallax D (x, y) within the region (x_(topleft),y_(topleft))−(x_(bottomright), y_(bottomright)) of the main subject asexpressed in Expression 1.

$\begin{matrix}{{D\lbrack{main}\rbrack} = \frac{\overset{ybottomright}{\sum\limits_{ytopright}}{\overset{xbottomright}{\sum\limits_{xtopright}}{D\left( {x,y} \right)}}}{\left( {{xbottomright} - {xtopleft} + 1} \right) \times \left( {{ybottomright} - {ytopleft} + 1} \right)}} & \left\lbrack {{Expression}\mspace{14mu} 1} \right\rbrack\end{matrix}$

Next, the right viewpoint image R is shifted in the horizontal directionby the parallax D [main] so that the parallax of the main subjectbetween the left viewpoint image L and the right viewpoint image Rbecomes 0 (Step S6). FIG. 2D is a view illustrating the left viewpointimage L and the right viewpoint image R after the shift. Coordinates ofthe corresponding point R on the right viewpoint image R after the shiftare (x_(L), y_(L)).

On the basis of the parallax D [main] which is the shift amount of theright viewpoint image R, the per-pixel parallax acquired in Step S4 isupdated (Step S7). That is, the per-pixel parallax after the update isexpressed as newD (x, y)=D (x, y)−D [main]. This newD (x, y) correspondsto a difference diff (x, y) between the per-pixel parallax D (x, y) andthe parallax of the main subject. A correction amount Factor2 of theper-pixel parallax is set in accordance with this diff (x, y) (Step S8).That is, such an expression that Factor2 (x, y)=Table (Diff (x, y)) isobtained.

Here, a function Table (x) is described.

FIG. 3A to FIG. 3D are graphs each illustrating the function Table (x)for setting the correction amount of the parallax for suppressing astereoscopic effect in accordance with a difference to the parallax ofthe main subject, in each of which the horizontal axis indicates adifference Diff (x, y) to the parallax of the main subject, and thevertical axis indicates the correction amount of each parallax.

FIG. 3A is a graph illustrating the function Table (x) when thecorrection is not performed. As illustrated in this figure, in the casewhere the per-pixel correction is not performed, the correction amountFactor2 is always set to 1.

FIG. 3B illustrates the function Table (x) when the parallax is moresuppressed (the stereoscopic effect is more suppressed) with an increasein deviation from the parallax of the main subject. The parallax at thecorresponding pixel is suppressed by setting the correction amountFactor2 to a value smaller than 1 in this way.

In addition, FIG. 3C illustrates the function Table (x) when thecorrection is not performed in a range of x_(min)≦x≦x_(max) and theparallax is suppressed in a portion smaller than x_(min) and a portionlarger than x_(max). Values of x_(min) and x_(max) may be decided asappropriate.

In addition, FIG. 3D illustrates the function Table (x) when thecorrection is not performed in the range of x_(min)≦x≦x_(max) and amagnification factor of the parallax is more reduced in proportion to anincrease in deviation from the parallax of the main subject in theportion smaller than x_(min) and the portion larger than x_(max).

As described above, the per-pixel correction amount Factor2 may be givento the per-pixel parallax by a linear or nonlinear expression, or may begiven thereto by a table. In addition, here, symmetric positive andnegative values are set to diff (x, y)=0, but asymmetric values may beset thereto.

In addition, FIG. 4 are graphs each illustrating the function Table (x)for setting the correction amount of the parallax for enhancing thestereoscopic effect in accordance with the difference to the parallax ofthe main subject.

FIG. 4A is a graph illustrating the function Table (x) when thecorrection is not performed similarly to FIG. 3A, and FIG. 4Billustrates the function Table (x) when the parallax is more increased(the stereoscopic effect is more enhanced) with a decrease in deviationfrom the parallax of the main subject. The parallax at the correspondingpixel is increased by setting the correction amount Factor2 to a valuelarger than 1 in this way.

In addition, FIG. 4C illustrates the function Table (x) when theparallax is more increased with a decrease in deviation from theparallax of the main subject in x_(min)≦x≦x_(max) and the correction isnot performed in the portion smaller than x_(min) and the portion largerthan x_(max), and FIG. 4D illustrates the function Table (x) when theparallax is uniformly increased in x_(min)≦x≦x_(max) and themagnification factor of the parallax is more reduced in proportion to anincrease in deviation from the parallax of the main subject in theportion smaller than x_(min) and the portion larger than x_(max).

As described above, the per-pixel correction amount Factor2 (x, y) maybe set so as to enhance the stereoscopic effect in accordance with thedifference to the parallax of the main subject.

The left viewpoint image L is converted on the basis of: the correctionamount Factor2 (x, y) of the per-pixel parallax which is set asdescribed above; and the conversion factor Factor1 [i] of the per-imageparallax which is set in Step S2, whereby six multi-viewpoint images C0to C5 are generated (Step S9). That is, each pixel Cn (x, y) of themulti-viewpoint image Cn is calculated as follows from each pixel L (x,y) of the left viewpoint image L.Cn(x+Factor1[n]×Factor2(x,y)×newD(x,y)=L(x,y)  Equation 1

FIG. 2E is a view illustrating the multi-viewpoint images C0 to C5 thusgenerated, in which C2 to C4 are omitted. The parallax of the mainsubject on each of the multi-viewpoint images C0 to C5 is 0.

Next, the six multi-viewpoint images C0 to C5 are each shifted in thehorizontal direction so that the parallax of the main subject on themulti-viewpoint images C0 to C5 becomes a predetermined parallax D[target], whereby final multi-viewpoint images S0 to S5 are generated(Step S10). That is, each pixel Sn (x, y) of the final multi-viewpointimage Sn is calculated as follows from each pixel Cn (x, y) of themulti-viewpoint image Cn.Sn(x,y)=Cn(x−D[target]×Factor1[n]/Factor1[5],y)  Equation 2

FIG. 2F is a view illustrating the final multi-viewpoint images S0 to S5thus generated, in which S2 to S4 are omitted. The parallax of the mainsubject on each of the final multi-viewpoint images S0 to S5 isappropriately set.

A stereoscopic image for output is generated from the six finalmulti-viewpoint images S0 to S5 (Step S11). In this step, a processingmethod suited to each output apparatus, such as multi-viewpointlenticular image conversion and image conversion for a multi-viewpointliquid crystal barrier monitor, is adopted.

Lastly, the generated stereoscopic image is outputted to the outputapparatus (Step S12).

In this way, from the inputted two-viewpoint images, the multi-viewpointimages are generated by using the conversion factor of the per-imageparallax and the correction amount of the per-pixel parallax which isset in accordance with the difference to the parallax of the mainsubject, and the images are shifted on the basis of the target parallaxof the main subject, whereby the final multi-viewpoint images aregenerated, and as a result, it is possible to obtain the stereoscopicimage having the parallax of the entire image and the parallax of themain subject which are appropriate.

In the present embodiment, the parallax D (x, y) is calculated withreference to the left viewpoint image L, and on the basis of thecalculated parallax D (x, y), the final multi-viewpoint images S0 to S5are generated with reference to the left viewpoint image L, and it ispreferable that, according to the same procedure, the parallax D (x, y)be calculated with reference to the right viewpoint image R, and on thebasis of the calculated parallax D (x, y), the final multi-viewpointimages S5 to S0 be generated with reference to the right viewpoint imageR, and then the respective multi-viewpoint images generated withreference to the left viewpoint image L and the respectivemulti-viewpoint images generated with reference to the right viewpointimage R be composited with each other. It should be noted that themulti-viewpoint images C0 to C5 may be composited with reference to theright and left viewpoint images, and the final multi-viewpoint images S0to S5 may be generated on the basis of the composite multi-viewpointimages C0 to C5.

In addition, in the present embodiment, the per-pixel parallax isacquired, and accordingly the correction amount of the per-pixelparallax is calculated, alternatively, a per-region parallax may beacquired, and accordingly the correction amount of the per-regionparallax may be calculated. In addition, the correction amount of theper-image parallax may be calculated.

Second Embodiment

Here, description is given of an example in which the main subject isdetected from collateral information of the image, and the correctionamount Factor2 according to the detected main subject is set.

First, as illustrated in a flow chart of FIG. 5, when the paralleltwo-eye stereo image (two-viewpoint images) is inputted (Step S1), acorrection process mode which is set in advance by a user is determined(Step S21). In the case where the correction process mode is set to amanual mode, correction according to a user's instruction is performed(Step S22).

Meanwhile, in the case where the correction process mode is set to anautomatic mode, the photographing mode which is set to an image pick-upapparatus when the inputted two-viewpoint images are photographed by theimage pick-up apparatus is acquired from collateral information of theinputted two-viewpoint image data (Step S23).

Next, it is determined whether or not the acquired photographing mode isa person/portrait mode (Step S24). In the case where the acquiredphotographing mode is the person/portrait mode, the main subject can beconsidered to be a person, and hence main subject detection is performedby person recognition and face recognition (Step S25). In addition, thecorrection amount Factor2 for performing such correction as to enhancethe stereoscopic effect of the detected main subject is set (Step S26).

In this way, because the main subject on the image photographed in theperson/portrait mode is clear, such correction as to enhance thestereoscopic effect of the main subject is performed.

Meanwhile, in the case where the acquired photographing mode is not theperson/portrait mode, it is next determined whether or not the acquiredphotographing mode is a scenery mode (Step S27). In the case where theacquired photographing mode is the scenery mode, the main subjectdetection is performed by detecting a median value in the parallaxhistogram (Step S28), and the correction amount Factor2 for performingsuch correction as to suppress the stereoscopic effect of the peripheryis set (Step S29).

In this way, because the main subject on the image photographed in thescenery mode is unclear, the main subject region is detected by usingthe center-weighted parallax histogram. Further, the image photographedin the scenery mode has extremely large parallax distribution, and hencein order to prevent excessively strong parallax in the short-distanceview and the long-distance view from making stereoscopic viewimpossible, the correction is performed so as to suppress thestereoscopic effect of a region having a parallax which is significantlydifferent from that of the main subject.

It should be noted that, in the case where the photographing modeacquired from the collateral information of the image data is not anyone of the person/portrait mode and the scenery mode, the correction isnot performed (Step S30).

In this way, the main subject is detected from the collateralinformation of the image data, and Factor2 according to the detectedmain subject is set, whereby the correction suited to the main subjectcan be performed.

Third Embodiment

With reference to FIG. 6, a generation process for multi-viewpointimages according to a third embodiment is described. In the presentembodiment, multi-viewpoint images are generated by compressing theparallax in accordance with the conversion factor of the parallax, andin the case where a range of the parallax of the main subject on themulti-viewpoint images thus generated is smaller than a range of thetarget parallax, the correction amount is set so as to enhance thestereoscopic effect of the main subject.

Similarly to the first embodiment, when the two-viewpoint images areinputted (Step S1), the main subject is detected from the left viewpointimage L, and the region (x_(topleft), y_(topleft))−(x_(bottomright),y_(bottomright)) of the main subject within the image is acquired (StepS3), and with reference to the left viewpoint image L, the per-pixelparallax D (x, y) to the right viewpoint image R is acquired (Step S4),and the parallax D [main] of the main subject is acquired (Step S5).

Next, the right viewpoint image R is shifted in the horizontal directionby the parallax D [main] so that the parallax of the main subjectbetween the left viewpoint image L and the right viewpoint image Rbecomes 0 (Step S6), and the per-pixel parallax is updated (Step S7).

Here, a maximum value D [main] max and a minimum value D [main] min inthe parallax distribution of pixels contained in the region(x_(topleft), y_(topleft))−(x_(bottomright), y_(bottomright)) of themain subject on the left viewpoint image L are acquired (Step S31).

Further, a maximum value D [entirety] max and a minimum value D[entirety] min in the parallax distribution of all pixels excluding 10%of the image periphery of the left viewpoint image L are acquired (StepS32).

Next, on the basis of: a maximum value D [target entirety] max and aminimum value D [target entirety] min in predetermined target parallaxdistribution of the entire image; and the maximum value D [entirety] maxand the minimum value D [entirety] min in the parallax distribution ofall the pixels acquired in Step S32, the conversion factor Factor1 ofthe per-image parallax is set (Step S33).

The setting of the conversion factor is described with reference to aflow chart of FIG. 7.

First, it is determined whether or not D [target entirety] max≧D[entirety] max and D [target entirety] min≦D [entirety] min, that is,whether or not the parallax distribution of all the pixels falls withinthe target parallax distribution (Step S41). If yes, a variable tmp isset to 1, and similarly to the first embodiment, the conversion factorsFactor1 [0] to Factor1 [5] of the per-image parallax are set to 0, 0.2,0.4, 0.6, 0.8, and 1.0, respectively (Step S42).

In the case where the parallax distribution of all the pixels does notfall within the target parallax distribution, the variable tmp is set toa larger value of D [entirety] max/D [target entirety] max and D[entirety] min/D [target entirety] min. In addition, with the use of thevariable tmp, the conversion factors Factor1 [0] to Factor1 [5] of theper-image parallax are set to 0, 0.2/tmp, 0.4/tmp, 0.6/tmp, 0.8/tmp, and1.0/tmp, respectively (Step S43).

For example, in the case where the inputted two-viewpoint images haveparallax distribution illustrated in FIG. 8A, even if themulti-viewpoint images are generated at such parallaxes, the parallaxesare excessively large and thus make stereoscopic view impossible.Accordingly, the per-pixel parallax is divided by the variable tmp, tobe thereby converted into parallax distribution illustrated in FIG. 8B,so that the parallax distribution is caused to fall within the targetparallax distribution. According to the parallax distribution after theconversion, the conversion factor Factor1 of the per-image parallaxbetween the multi-viewpoint images is set.

In this way, the conversion factor Factor1 of the per-image parallax isappropriately set on the basis of the parallax distribution of all thepixels and the target parallax distribution of the entire image.

Next, on the basis of: a maximum value D [target main] max and a minimumvalue D [target main] min in predetermined target parallax distributionof the main subject; and the maximum value D [main] max and the minimumvalue D [main] min of the parallax distribution of the main subject (inthe case where the parallax distribution of all the pixels is correctedby Factor1, the parallax distribution of the main subject after thecorrection), the correction amount Factor2 of the per-pixel parallax isset (Step S34).

The setting of the correction amount is described with reference to aflow chart of FIG. 9.

First, it is determined whether or not D [target main] max−D [targetmain] min≦(D [main] max−D [main] min)/tmp, that is, whether or not theparallax range of the main subject is wider than the target parallaxrange (Step S51). In the case where the parallax range of the mainsubject is wider (YES in Step S51), it is determined that thestereoscopic effect of the main subject is sufficient, and thecorrection amount Factor2 of the per-pixel parallax is set to 1 (StepS52). That is, the correction of the per-pixel parallax is notperformed.

In addition, in the case where the target parallax range is wider (NO inStep S51), it is determined that the stereoscopic effect of the mainsubject is insufficient, and the correction amount Factor2 is set so asto increase the parallax of the main subject. Here, the correctionamount Factor2 is set according to the function Table (x) illustrated inFIG. 4D, so that xmin=D [main] min/tmp and xmax=D [main] max/tmp (StepS53). That is, Factor2 is set so that the parallax is uniformlyincreased in D [main] min/tmp to D [main] max/tmp, and the magnificationfactor of the parallax is more reduced in proportion to an increase indeviation from the parallax of the main subject in a portion smallerthan D [main] min and a portion larger than D [main] max/tmp.

For example, in the case of the parallax distribution of the mainsubject and the target parallax distribution of the main subjectillustrated in FIG. 8B, the target parallax distribution is wider, andhence the correction amount Factor2 of the per-pixel parallax is set asin Step S53.

In this way, on the basis of: the actual parallax distribution of themain subject; and the target parallax distribution of the main subject,the appropriate correction amount Factor2 of the per-pixel parallax isset.

On the basis of: the correction amount Factor2 (x, y) of the per-pixelparallax which is set as described above; and the conversion factorFactor1 [i] of the per-image parallax which is set in Step S42 or StepS43, the six multi-viewpoint images C0 to C5 are generated (Step S9).Further, the final multi-viewpoint images S0 to S5 are generated so thatthe parallax of the main subject becomes the predetermined parallax D[target] (Step S10).

Lastly, the stereoscopic image for output is generated from the sixfinal multi-viewpoint images S0 to S5 (Step S11), and the generatedstereoscopic image is outputted (Step S12).

In this way, in the case where the range of the parallax of the mainsubject after the compression according to the conversion factor of theparallax is narrower than the range of the target parallax of the mainsubject, the correction is performed so as to enhance the stereoscopiceffect of the main subject, whereby a stereoscopic image having anappropriate parallax of the main subject can be generated.

It should be noted that it is preferable to modify the target parallaxin accordance with an output size of the multi-viewpoint image.

For example, in the case where the preset target parallax distributionof the entire image and the preset target parallax distribution of themain subject are suited to a predetermined reference output size, it isconceivable to make such a definition that a modification factorFactor3=the output size/the reference output size and modify the targetparallax distribution according to the modification factor Factor3. Thatis, each of the maximum value D [target entirety] max and the minimumvalue D [target entirety] min in the target parallax distribution of theentire image and the maximum value D [target main] max and the minimumvalue D [target main] min in the target parallax distribution of themain subject is divided by the modification factor Factor3, to therebymodify the target parallax distribution, and the processing of Step S33and Step S34 is performed on the basis of the modified value.

The magnitude of the parallax at which stereoscopic view is not possibleis different depending on the size of an outputted image, and hence thetarget parallax is modified as described above in accordance with theoutput size of the multi-viewpoint image, whereby it is possible togenerate the multi-viewpoint image having an appropriate parallax inaccordance with the size of the outputted image.

Fourth Embodiment

With reference to FIG. 10, a generation process for multi-viewpointimages according to a fourth embodiment is described. In the presentembodiment, the per-pixel correction amount Factor2 is set so as toplanarize parallax gradations within the image.

The processing from Step S1 to Step S7 is the same as the above, andhence description thereof is omitted.

Next, a parallax histogram is created from the per-pixel parallaxacquired in Step S4 (Step S61), and the per-pixel correction amountFactor2 is set in accordance with the created histogram (Step S62). Inthis step, Factor2 is set on the basis of the following expression.

$\begin{matrix}{{{Factor}\; 2(D)} = {\frac{1}{{Hist}(0)} \times \frac{\sum\limits_{x = 0}^{D}{{Hist}(x)}}{{D} + 1}}} & \left\lbrack {{Expression}\mspace{14mu} 2} \right\rbrack\end{matrix}$

For example, in the case of parallax distribution illustrated in FIG.11A, the per-pixel correction amount Factor2 (F2 (D)) calculated on thebasis of [Expression 2] is set as illustrated in FIG. 11B.

The subsequent processing from Step S10 to Step S12 is the similar tothose of the above embodiment, and hence description thereof is omitted.

In this way, the correction amount of the parallax is increased toenhance the stereoscopic effect in a portion having higher frequenciesin the parallax histogram, and the correction amount of the parallax isreduced to suppress the stereoscopic effect in a portion having lowerfrequencies, whereby the contrast of the parallax is made uniform.Accordingly, it becomes easier for a user to discriminate long-distanceview and short-distance view from each other and to discriminate andrecognize the stereoscopic effect.

Fifth Embodiment

With reference to FIG. 12 and FIG. 13, a generation process formulti-viewpoint images according to a fifth embodiment is described. Inthe present embodiment, the main subject region is set in accordancewith the photographing mode at the time of photographing thetwo-viewpoint images.

When the two-viewpoint images are inputted (Step S1), the conversionfactor Factor1 of the per-image parallax is set (Step S2), and theper-pixel parallax D (x, y) is acquired (Step S4).

Next, the photographing mode which has been set to the image pick-upapparatus when the inputted two-viewpoint images has been photographedby the image pick-up apparatus is acquired from the collateralinformation of the inputted two-viewpoint image data (Step S71), and itis determined whether or not the acquired photographing mode is theperson/portrait mode (Step S72).

In the case where the person/portrait mode has been set (YES in StepS72), an image central portion of a plurality of regions obtained byvertically dividing the left viewpoint image L is set as the mainsubject region (Step S73). For example, as illustrated in FIG. 14A, theleft viewpoint image L is vertically divided equally into three, and acentral region A1 is set as the main subject region.

An average value of parallaxes in the main subject region is calculated,and is defined as the parallax D [main] of the main subject (Step S74).Further, the right viewpoint image R is shifted in the horizontaldirection by the parallax D [main] so that the parallax of the mainsubject between the left viewpoint image L and the right viewpoint imageR becomes 0 (Step S75), and the per-pixel parallax acquired in Step S4is updated in accordance with the parallax D [main] which is the shiftamount (Step S76).

Next, a distance L (x, y) on the image in the horizontal direction fromthe main subject region is acquired per pixel (Step S77), and theper-pixel correction amount Factor2 (x, y)=f (L (x, y)) is set inaccordance with the distance (Step S78).

For example, in the example of FIG. 14A, Factor2 (x, y) is set so thatthe parallax is uniformly increased in the main subject region A1, andthe magnification factor of the parallax is more reduced in proportionto an increase in deviation in the horizontal direction from the mainsubject region in regions to the right and left of the main subjectregion A1.

On the basis of: the per-pixel correction amount Factor2 (x, y) thusset; and the per-image conversion factor Factor1 set in Step S2, theleft viewpoint image L is converted, and the six multi-viewpoint imagesC0 to C5 are generated (Step S79).

The subsequent processing from Step S10 to Step S12 is the same as theabove, and hence description thereof is omitted.

Meanwhile, in the case where the acquired photographing mode is not theperson/portrait mode (NO in Step S72), the processing proceeds to a flowchart of FIG. 13, it is determined whether or not the acquiredphotographing mode is the scenery mode (Step S81).

In the case where the acquired photographing mode is not the scenerymode (NO in Step S81), the per-pixel correction is not performed, theleft viewpoint image L is converted on the basis of the per-imageconversion factor Factor1, and the six multi-viewpoint images C0 to C5are generated (Step S82).

In the case where the acquired photographing mode is the scenery mode(YES in Step S72), a central portion of a plurality of regions obtainedby horizontally dividing the image is set as the main subject region(Step S83). For example, as illustrated in FIG. 14B, the left viewpointimage L is horizontally divided equally into three, and a central regionA2 is set as the main subject region.

An average value of parallaxes in the main subject region is calculated,and is defined as the parallax D [main] of the main subject (Step S84).Further, the right viewpoint image R is shifted in the horizontaldirection by the parallax D [main] so that the parallax of the mainsubject between the left viewpoint image L and the right viewpoint imageR becomes 0 (Step S85), and the per-pixel parallax acquired in Step S4is updated in accordance with the parallax D [main] which is the shiftamount (Step S86).

Next, a distance L2 (x, y) on the image in the vertical direction fromthe main subject region is acquired per pixel (Step S87), and theper-pixel correction amount Factor2 (x, y)=f (L2 (x, y)) is set inaccordance with the distance (Step S88).

For example, in the example of FIG. 14B, Factor2 (x, y) is set so thatthe correction is not performed in the main subject region A2, and theparallax is more suppressed in proportion to an increase in deviation inthe vertical direction from the main subject region in regions above andbelow the main subject region A2.

On the basis of: the per-pixel correction amount Factor2 (x, y) thusset; and the per-image conversion factor Factor1 set in Step S2, theleft viewpoint image L is converted, and the six multi-viewpoint imagesC0 to C5 are generated (Step S89).

In this way, the photographing mode at the time of photographing thetwo-viewpoint images is acquired from the collateral information, andthe main subject region is estimated on the basis of the photographingmode, whereby an appropriate parallax amount can be given withoutcalculating the parallax distribution.

Sixth Embodiment

With reference to FIG. 15, a generation process for multi-viewpointimages according to a sixth embodiment is described. In the presentembodiment, scenes on the inputted two-viewpoint images are analyzed, tothereby discriminate short-distance view, middle-distance view, andlong-distance view from one another, and the middle-distance view regionis set as the main subject.

When the two-viewpoint images are inputted (Step S1), the conversionfactor Factor1 of the per-image parallax is set (Step S2), and theper-pixel parallax D (x, y) is acquired (Step S4).

Next, scenes on the inputted two-viewpoint image data are analyzed,classification into the short-distance view, the middle-distance view,and the long-distance view is performed per region (Step S81), and themiddle-distance view region is set as the main subject (Step S82). Inthe scene analysis, for example, the left viewpoint image L is dividedinto a plurality of regions, and a brightness histogram is created perdivided region, and then, a region having wide brightness distribution,that is, a region having high contrast is classified into theshort-distance view, and a region having narrow brightness distribution,that is, a region having low contrast is classified into thelong-distance view.

Next, on the basis of the per-pixel parallax D (x, y) acquired in StepS4, an average value of parallaxes in the main subject region set inStep S81 is calculated, and the parallax D [main] of the main subject isacquired (Step S5).

Further, the right viewpoint image R is shifted in the horizontaldirection by the parallax D [main] so that the parallax of the mainsubject between the left viewpoint image L and the right viewpoint imageR becomes 0 (Step S6), and the per-pixel parallax is updated (Step S7).

Then, the per-pixel correction amount Factor2 is set to each of theshort-distance view region, the middle-distance view region, and thelong-distance view region (Step S83).

For example, it is conceivable to set Factor2 (x, y) so that theparallax is uniformly increased in the middle-distance view region, andthe correction is not performed in the short-distance view region andthe long-distance view region, and it is also conceivable to set Factor2(x, y) so that the correction is not performed in the middle-distanceview region, and the parallax is suppressed in the short-distance viewregion and the long-distance view region.

The subsequent processing from Step S9 to Step S12 is similar to theabove, and hence description thereof is omitted.

In this way, the short-distance view, the middle-distance view, and thelong-distance view are discriminated by the scene analysis, and then,even in the case where the per-pixel parallax D (x, y) is erroneouslycalculated by corresponding point detection, the middle-distance viewregion can be appropriately set as the main subject region.

Other Modified Examples

In the first and other embodiments, the conversion factors Factor1 (0)to Factor1 (5) of the per-image parallax are set to 0, 0.2, 0.4, 0.6,0.8, and 1.0, respectively, but may be set at irregular intervals, forexample, to 0, 0.1, 0.3, 0.7, 0.9, and 1.0.

When the conversion factors are set as described above, for example, theparallax between the final multi-viewpoint images S0-S1 and the parallaxbetween the final multi-viewpoint images S4-S5 are compressed todiminish the stereoscopic effect, so that image failure is moreavoidable. Further, the stereoscopic effect between the viewpoint imagesS1-S2 and the stereoscopic effect between the viewpoint images S3-S4 canbe reproduced as they are. Still further, the stereoscopic effectbetween the viewpoint images S2-S3 can be enhanced.

In this way, the conversion factor Factor1 of the per-image parallax maybe set at regular intervals, or may be set so that a difference inparallax conversion factor between images in the central portion islarger than a difference in parallax conversion factor between images atboth end portions.

In addition, the conversion factor Factor1 of the per-image parallax andthe correction amount Factor2 of the per-pixel parallax are calculated,and then the respective viewpoint images C0 to C5 are generated withreference to the left viewpoint image L, but the respective viewpointimages may be generated with reference to the central portion of theviewpoints of the generated multi-viewpoint images.

For example, instead of Equation 1, the use of Equation 3 given belowmakes it possible to define the central portion of the respectiveviewpoints (in the case of the six multi-viewpoint images, a portionbetween the viewpoint images C2 and C3) as the reference of therespective viewpoint images.Cn(x+Factor1(n)×(Factor2(x,y)−1.0)/2×newD(x,y),y)=L(x,y)  Equation 3

If the left viewpoint image L is used as the reference, there is apossibility that image distortion caused by the per-pixel parallax whichis more enhanced/suppressed toward the right viewpoint image becomeslarger, in contrast, if the central portion of the respective viewpointsis used as the reference, the image distortion can be distributedequally to the right and left, so that natural multi-viewpoint imagescan be generated.

<Configuration of Image Processing Apparatus>

FIG. 16 is a block diagram illustrating an image processing apparatus 10for implementing the first to sixth embodiments. The image processingapparatus 10 is configured by, for example, a personal computer and aworkstation. The image processing apparatus 10 includes an image inputunit 11, an image output unit 12, a processing method instruction unit13, a parallax acquisition unit 15, a parallax conversion factor settingunit 17, a parallax conversion factor correction unit 18, an imageconversion unit 20, and a stereoscopic image generation unit 23.

The image input unit 11 receives the left viewpoint image L and theright viewpoint image R (two-viewpoint images) which are photographed asa stereo image, and corresponds to, for example: an image readingapparatus which reads a multi-picture file (MP file) in whichmulti-viewpoint images for a stereoscopic image are coupled to eachother, from a recording medium which stores therein the MP file; and anapparatus which acquires the MP file via a network.

The image output unit 12 outputs the generated stereoscopic image, andcorresponds to: a printing apparatus for a stereoscopic photographicprint having a surface on which a lenticular lens sheet is attached; aparallax barrier monitor; and the like.

The processing method instruction unit 13 is an operation unit forsetting the correction process mode of the automatic mode/the manualmode which is determined in Step S21 of FIG. 5, and for setting theper-image conversion factor Factor1 and the per-pixel correction amountFactor2. In addition, an output size instruction unit 14 is an operationunit for giving a size instruction of a stereoscopic image to beoutputted.

These units are configured by a keyboard and a pointing device. Itshould be noted that the output size instruction unit 14 may beconfigured to automatically acquire the output size of the image outputunit 12.

The parallax acquisition unit 15 acquires the per-pixel or per-regionparallax D (x, y) between the left viewpoint image L and the rightviewpoint image R received by the image input unit 11, and is used in,for example, Step S4 of FIG. 1.

A histogram creation unit 16 creates a parallax histogram on the basisof the per-pixel or per-region parallax D (x, y) acquired by theparallax acquisition unit 15, and is used in, for example, Step S61 ofFIG. 10.

The parallax conversion factor setting unit 17 sets the conversionfactors Factor1 (0) to Factor1 (5) of the parallaxes between therespective viewpoint images. On the basis of an input from theprocessing method instruction unit 13, it sets the conversion factors atregular intervals, for example, to 0, 0.2, 0.4, 0.6, 0.8, and 1.0, andsets the conversion factors so that a difference in parallax conversionfactor between images in the central portion is larger than a differencein parallax conversion factor between images at both end portions, forexample, to 0, 0.1, 0.3, 0.7, 0.9, and 1.0.

The parallax conversion factor correction unit 18 decides the correctionamounts of the respective viewpoint images so that the correction amountof the parallax of an image to be used as the reference becomes 0 orsmallest. For example, the use of Equation 1 makes it possible to obtainthe correction amount with reference to the left viewpoint image L, andthe use of Equation 3 makes it possible to define the portion betweenthe viewpoint images C2 and C3 as the reference of the respectiveviewpoint images.

A correction method selection unit 19 selects a calculation method forthe correction amount Factor2 in accordance with a detection result of aphotographing mode detection unit 28 to be described later, and is usedin, for example, Step S24 and Step S27 of FIG. 5.

The image conversion unit 20 includes a parallax→image conversion unit21 and an image shift unit 22.

The parallax→image conversion unit 21 converts the image on the basis ofthe conversion factor Factor1 of the per-image parallax and thecorrection amount Factor2 of the per-pixel parallax, to thereby generatethe multi-viewpoint images, and is used in, for example, Step S9 of FIG.1.

In addition, the image shift unit 22 shifts the multi-viewpoint imagesin the horizontal direction in accordance with the parallax D [target],and is used in, for example, Step S10 of FIG. 1.

The stereoscopic image generation unit 23 generates the stereoscopicimage for output on the basis of the generated final multi-viewpointimages, and is used in, for example, Step S11 of FIG. 1.

A parallax comparison unit 24 compares the per-pixel parallax with theparallax of the main subject, and is used in, for example, Step S8 ofFIG. 1.

An image analysis unit 25 includes a header analysis unit 26 and animage data analysis unit 29.

The header analysis unit 26 acquires image information from thecollateral information recorded in a header part of the image file, andincludes: an image size acquisition unit 27 which acquires an imagesize; and a photographing mode detection unit 28 which acquires thephotographing mode which is set at the time of photographing the images.The detection result of the photographing mode detection unit 28 is usedin, for example, Step S24 and Step S27 of FIG. 5.

In addition, the image data analysis unit 29 analyzes image data on thebasis of a pixel value of each pixel recorded in an image main part ofthe image file, and includes: a scene analysis unit 30 which analyzes aphotographed scene; and a main subject detection unit 31 which detectsthe main subject within the image. An analysis result of the sceneanalysis unit 30 is used in, for example, Step S81 of FIG. 15. Inaddition, the main subject detection unit 31 includes a unit whichdetects a face of a person within the image as the main subject, and adetection result thereof is used in, for example, Step S3 of FIG. 1.

A main subject setting unit 32 sets the main subject region within theimage on the basis of the detection result of the photographing modedetection unit 28 and the analysis result of the scene analysis unit 30,and is used in, for example, Step S73 of FIG. 12 and Step S83 of FIG.13.

A main subject target parallax holding unit 33 holds the parallax D[target] which is the target parallax of the main subject on each finalmulti-viewpoint image, and the parallax D [target] is used in, forexample, Step S10 of FIG. 1.

A main subject target parallax distribution holding unit 34 holds thetarget parallax distribution of the main subject, and an entire targetparallax distribution holding unit 35 holds the target parallaxdistribution of the entire image. These target parallax distributionsare used in, for example, Step S53 of FIG. 9 and Step S43 of FIG. 7.

A target value modification unit 36 modifies the target parallax betweenthe multi-viewpoint images, and for example, calculates the modificationfactor Factor3 in accordance with an input from the output sizeinstruction unit 14, and modifies the target parallaxes held by the mainsubject target parallax holding unit 33 and the main subject targetparallax distribution holding unit 34 on the basis of the modificationfactor Factor3.

The first to sixth embodiments described above can be implemented by theimage processing apparatus 10 thus configured.

It should be noted that, although every generation process for themulti-viewpoint images is realized by hardware herein, the generationprocess can also be realized as a multi-viewpoint image generationprogram for controlling the image processing apparatus 10.

REFERENCE SIGNS LIST

10 . . . image processing apparatus, 11 . . . image input unit, 12 . . .image output unit, 13 . . . processing method support unit, 15 . . .parallax acquisition unit, 16 . . . histogram creation unit, 17 . . .parallax conversion factor setting unit, 18 . . . parallax conversionfactor correction unit, 20 . . . image conversion unit, 21 . . .parallax→image conversion unit, 22 . . . image shift unit, 23 . . .stereoscopic image generation unit, 25 . . . image analysis unit, 26 . .. header analysis unit, 29 . . . image data analysis unit, 32 . . . mainsubject setting unit

The invention claimed is:
 1. An image processing apparatus comprising:an image input unit configured to receive two-viewpoint images having aparallax therebetween; a parallax acquisition unit configured to acquirea per-pixel or per-region parallax between the two-viewpoint images; amain subject detection unit configured to detect a main subject on thetwo-viewpoint images; a parallax acquisition unit configured to acquirea parallax of the main subject; a setting unit configured to set aconversion factor of the parallax in accordance with respectiveviewpoint positions of multi-viewpoint images to be generated; acorrection unit configured to correct the conversion factor of theparallax per pixel, per region, or per image on a basis of the parallaxof the main subject; a multi-viewpoint image generation unit configuredto convert at least one image of the two-viewpoint images in accordancewith the corrected conversion factor of the parallax, to therebygenerate the multi-viewpoint images; an image adjustment unit configuredto shift the two-viewpoint images or the multi-viewpoint images in ahorizontal direction so that the parallax of the main subject on themulti-viewpoint images becomes a parallax appropriate for stereoscopicview; and a stereoscopically-displayed image generation unit configuredto generate a stereoscopically-displayed image on a basis of themulti-viewpoint images, wherein the correction unit corrects theconversion factor of the parallax per pixel, per region, or per image ona basis of a difference between the per-pixel or per-region parallax andthe parallax of the main subject, the image processing apparatus furthercomprising: a photographing mode detection unit configured to detect aphotographing mode set at a time of photographing the two-viewpointimages, from collateral information of the two-viewpoint images; and aselection unit configured to select, in accordance with the detectedphotographing mode, whether the correction unit corrects the conversionfactor of the parallax so that the parallax is more suppressed in apixel or region having larger difference between parallaxes or correctsthe conversion factor of the parallax so that the parallax is moreincreased in a pixel or region having smaller difference between theparallaxes.
 2. The image processing apparatus according to claim 1,further comprising a holding unit configured to hold target parallaxdistribution of an entire image and target parallax distribution of themain subject, wherein: the setting unit sets the conversion factor ofthe parallax so that parallax distribution of the entire image of themulti-viewpoint images to be generated satisfies the target parallaxdistribution of the entire image; and the correction unit corrects theconversion factor of the parallax per pixel, per region, or per image sothat parallax distribution of the main subject on the multi-viewpointimages to be generated satisfies the target parallax distribution of themain subject.
 3. The image processing apparatus according to claim 1,further comprising: an output unit configured to output the generatedstereoscopically-displayed image at a predetermined size; and amodification unit configured to modify the target parallax distributionof the entire image and the target parallax distribution of the mainsubject in accordance with the predetermined size.
 4. The imageprocessing apparatus according to claim 1, wherein the setting unitsets, in a case where the multi-viewpoint images to be generated arefour or more, the conversion factor of the parallax so that a differencebetween viewpoint positions in a central portion is larger than adifference between viewpoint positions at both end portions.
 5. Theimage processing apparatus according to claim 1, wherein: the correctionunit corrects the conversion factor of the parallax per pixel, perregion, or per image so that the conversion factor of the parallax ofone image of the two-viewpoint images becomes 0; and the multi-viewpointimage generation unit converts the image whose conversion factor of theparallax is 0, to thereby generate the multi-viewpoint images.
 6. Theimage processing apparatus according to claim 1, wherein the correctionunit corrects the conversion factor of the parallax per pixel, perregion, or per image so that the conversion factor of the parallax of animage at a central viewpoint position, of the multi-viewpoint images tobe generated, becomes smallest.
 7. An image processing apparatuscomprising: an image input unit configured to receive two-viewpointimages having a parallax there between; a parallax acquisition unitconfigured to acquire a per-pixel or per-region parallax between thetwo-viewpoint images; a main subject detection unit configured to detecta main subject on the two-viewpoint images; a parallax acquisition unitconfigured to acquire a parallax of the main subject; a setting unitconfigured to set a conversion factor of the parallax in accordance withrespective viewpoint positions of multi-viewpoint images to begenerated; a correction unit configured to correct the conversion factorof the parallax per pixel, per region, or per image on a basis of theparallax of the main subject; a multi-viewpoint image generation unitconfigured to convert at least one image of the two-viewpoint images inaccordance with the corrected conversion factor of the parallax, tothereby generate the multi-viewpoint images; an image adjustment unitconfigured to shift the two-viewpoint images or the multi-viewpointimages in a horizontal direction so that the parallax of the mainsubject on the multi-viewpoint images becomes a parallax appropriate forstereoscopic view; a stereoscopically-displayed image generation unitconfigured to generate a stereoscopically-displayed image on a basis ofthe multi-viewpoint images; a parallax histogram acquisition unitconfigured to acquire a histogram of the per-pixel or per-regionparallax, wherein the correction unit corrects the conversion factor ofthe parallax per pixel or per region so as to make parallax gradationsconstant in accordance with frequencies in the histogram.
 8. The imageprocessing apparatus according to claim 7, wherein the setting unitsets, in a case where the multi-viewpoint images to be generated arefour or more, the conversion factor of the parallax so that a differencebetween viewpoint positions in a central portion is larger than adifference between viewpoint positions at both end portions.
 9. Theimage processing apparatus according to claim 7, wherein: the correctionunit corrects the conversion factor of the parallax per pixel, perregion, or per image so that the conversion factor of the parallax ofone image of the two-viewpoint images becomes 0; and the multi-viewpointimage generation unit converts the image whose conversion factor of theparallax is 0, to thereby generate the multi-viewpoint images.
 10. Theimage processing apparatus according to claim 7, wherein the correctionunit corrects the conversion factor of the parallax per pixel, perregion, or per image so that the conversion factor of the parallax of animage at a central viewpoint position, of the multi-viewpoint images tobe generated, becomes smallest.
 11. An image processing apparatuscomprising: an image input unit configured to receive two-viewpointimages having a parallax there between; a parallax acquisition unitconfigured to acquire a per-pixel or per-region parallax between thetwo-viewpoint images; a main subject detection unit configured to detecta main subject on the two-viewpoint images; a parallax acquisition unitconfigured to acquire a parallax of the main subject; a setting unitconfigured to set a conversion factor of the parallax in accordance withrespective viewpoint positions of multi-viewpoint images to begenerated; a correction unit configured to correct the conversion factorof the parallax per pixel, per region, or per image on a basis of theparallax of the main subject; a multi-viewpoint image generation unitconfigured to convert at least one image of the two-viewpoint images inaccordance with the corrected conversion factor of the parallax, tothereby generate the multi-viewpoint images; an image adjustment unitconfigured to shift the two-viewpoint images or the multi-viewpointimages in a horizontal direction so that the parallax of the mainsubject on the multi-viewpoint images becomes a parallax appropriate forstereoscopic view; a stereoscopically-displayed image generation unitconfigured to generate a stereoscopically-displayed image on a basis ofthe multi-viewpoint images; a photographed scene recognition unitconfigured to recognize a photographed scene on the two-viewpointimages; and a main subject setting unit configured to set a main subjectregion in accordance with the photographed scene, wherein the correctionunit corrects the conversion factor of the parallax per region or perpixel in accordance with a distance on the image from the main subjectregion.
 12. The image processing apparatus according to claim 11,wherein: the photographed scene recognition unit includes aphotographing mode detection unit configured to detect a photographingmode set at a time of photographing the two-viewpoint images, fromcollateral information of the two-viewpoint images; the main subjectregion setting unit sets, in a case where the detected photographingmode is a person mode, a central region of a plurality of regionsobtained by vertically dividing the image, as the main subject region;and the correction unit corrects the conversion factor of the parallaxper region or per pixel in accordance with a distance in the horizontaldirection from the main subject region.
 13. The image processingapparatus according to claim 11, wherein: the photographed scenerecognition unit includes a photographing mode detection unit whichdetects a photographing mode set at a time of photographing thetwo-viewpoint images, from collateral information of the two-viewpointimages; the main subject region setting unit sets, in a case where thedetected photographing mode is a scenery mode, a central region of aplurality of regions obtained by horizontally dividing the image, as themain subject region; and the correction unit corrects the conversionfactor of the parallax per region or per pixel in accordance with adistance in the vertical direction from the main subject region.
 14. Theimage processing apparatus according to claim 11, wherein: thephotographed scene recognition unit includes an extraction unitconfigured to extract a short-distance view region, a middle-distanceview region, and a long-distance view region from the two-viewpointimages; and the correction unit corrects the conversion factor of theparallax per region or per pixel in accordance with the respectiveextracted regions.
 15. The image processing apparatus according to claim11, wherein the setting unit sets, in a case where the multi-viewpointimages to be generated are four or more, the conversion factor of theparallax so that a difference between viewpoint positions in a centralportion is larger than a difference between viewpoint positions at bothend portions.
 16. The image processing apparatus according to claim 11,wherein: the correction unit corrects the conversion factor of theparallax per pixel, per region, or per image so that the conversionfactor of the parallax of one image of the two-viewpoint images becomes0; and the multi-viewpoint image generation unit converts the imagewhose conversion factor of the parallax is 0, to thereby generate themulti-viewpoint images.
 17. The image processing apparatus according toclaim 11, wherein the correction unit corrects the conversion factor ofthe parallax per pixel, per region, or per image so that the conversionfactor of the parallax of an image at a central viewpoint position, ofthe multi-viewpoint images to be generated, becomes smallest.